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ABSTRACT 

Four groups of people were requested to review the 
Rand Corporation's design for a longitudinal study of school 
desegregation. These groups can be labeled as Desegregation and 
Minority Student Education Experts, Distinguished Social Scientists, 
Federal Officals Responsible for Desegregation Research, and U.S. 
Commission on Civil Rights (USCCR) Staff Members. In late August, 
1974, letters were sent to prospective reviewers asking them to 
participate. Twenty-two of the 25 prospective reviewers outside of 
USCCR ugroed. Nineteen of the 22 outside reviewers submitted 
comments. One additional outside reviewer was contacted and submitted 
his commentfs. This summary is based on the feedback provided by those 
26 reviewers* When the study design documents were sent to the 
reviewers, an evaluation questionnaire was attached. The reviewers 
were asked to answer the 27 multiple choice questions, to add any 
needed clarifying comments in the spaces provided after each 
question, and to write up to six single spaced typed pages of 
additouai connertr addressing any issues they thought to be 
important. The responses of each respondent to the 27 questions are 
given, as ar*» the average rating for each reviewer, for each group, 
and for all groups together. Those results are briefly discussed and 
then the narrative comments made by each reviewer are summarized. 
(Author/JM) 
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INTRODUCTION 



Four groups of people were requested to review the Rand Corporation s 
design for a longitudinal study of school desegregation. These groups 
can be labeled as Desegregation and Minority Student Education Experts, 
Distinguished Social Scientists, Federal Officials Responsible for 
Desegregation Research, and U. S. Commission on Civil Rights (USCCR; 
Staff Members. These will be referred to in this discussion as groups 
I to IV, respectively. 

In late August, 1974, letters were sent to prospective reviewers asking 
them to participate in the review of the Rand study design. It was 
explained that the task would probably take about 3 full days of work, 
and persons who were not employees of the Federal Government were offered 
$400 in remuneration. Twenty-two of the 25 prospective reviewers out- 
side of USCCR agreed to try to find the time to review the documents. 
They were mailed the document on September 6, and asked to return their 
comments by October 3. Nineteen of the 22 outside reviewers returned 
comments, as did 6 USCCR staff members. One additional outside reviewer 
was contacted about a month later and later submitted his comments. 
This summary is based on the feedback provided by those 26 reviewers. 

At the time of the initial invitation, each prospective reviewer had 
been sent a brief history of the events which led USCCR to contract 
with Rand for the development of this study design, and of the feedback 
given to Rand by various review panels and the USCCR staff over the 
course of the design development. 

When the study design documents were sent to the reviewers, an evaluation 
questionnaire was attached. The reviewers were asked to answer the 27 
multiple choice questions (see attachment A ), to add any needed 
clarifying comments in the spaces provided after each question, and to 
write up to 6 pages (single spaced typing) of additional comments addressing 
any issues they thought to be important. 

The responses of each respondent to the 27 questions are given in 
at tachment B . The average rating for each reviewer, for each S^^oup, 
^ and for all groups together, are given in attachment C. Those results 
^ will bo briefly discussed below and then the narrative comments made 
by each reviewer will be summarized. 
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Before discussing the responses to the multiple choice evaluation question- 
naire there Is an Important consideration which needs to bo noted. The 
questionnaire asked the reviewers to evaluate various aspects of the 
Study design on a four point scale from quite good to quite poor, but 
did not suggest a standard against which this judgment should be made. 
It appears that threii different standards were used: 1) the level of 
quality generally manifest by previous desegregation research, 2) the 
level of quality which is possible, given the present state of the art 
In respect to social science evaluations, and 3) the level of quality 
necessary for making unambiguous inferences for at least some Ir.iportant , 
coherent sets of policy relevant questions. These standards are increasingly 
severe, and any person who primarily relied on the first one would certainly 
have rated the design much higher than if using the third one. It might 
have been possible and desirable to ask all reviewers to use t.ie same 
standard, but this did not occur co the staff at the time the evaluation 
questionnaire was written. In any case, it Is safe to say chat If the 
reviewers had been asked to rely primarily on the first standard the 
ratings would have been higher (but to an unknown extent) than they are, 
and if the reviewers had been asked to use the third standard the ratings 
would be lower (but also to an unknown extent). IL appears from the 
written comments that most reviewers relied primarily on the second 
standard, and that the other two standards were each used by a few of 
the reviewers. 

RESULTS OF THE EVALUATIVE QUESTIONNAIRE 

For purposes of tabulation, the four possible ratings were coded as 
follows: "quite good" as 1.0, "moderately good" as 2.0, "moderately- 
poor" as 3.0, "quite poor" as 4.0. The overall average rating was r..38, 
somewhat below "moderately good", but slightly above the midpoint in the 
rating scale. The average rating of the most favorable reviewer was 1.3, 
and v;as 3.5 for the least favorable reviewer. 

The most favorable ratings were given to the following factors: how 
well the design makes use of previous school desegregation research, 
the feasibility of implementing the non-experimental design option, and 
the ethicalness of both design options. The least favorable ratings? were 
In respect to: how well the design states questions or hypotheses 
specifically enough to be effectively and efficiently researched, the 
feasibility of implementing the experimental design option, the apparent 
likelihood of the Instruments yielding valid measures of the variables of 
Interest in the study, and the extent to which the instruments cover an 
flppropriaLti set of. behavioral variables. 

There nrc fiome dif i'^erences between the 4 groups of reviewers. The 
Dlstlnquishod Social Scientists gave the design the highest average 
rating (1.96) and the USCCR Staff gave it the lowest average rating (2.82). 
The Federal Officials responsible for Deseg/'egation Research and the 
Desegregation and Minority Student Education Experts '/ere in the middle 
with avcrogc ratings of tAQ and 2.66 respectively. 



It should be noted that these are unweighted averages and thus presume 
that each question is as important as all othurs aiul tlml each person 
was equally competent and careful in answering those- qucistions for which 
they indicated a rating. This probably is not entirely true, but any 
system of weights would involve o:her presumptions whicli also would be 
questionable , 

ily 7 of the 26 reviewers suggested that USCCR directly proceed with 
soliciting funding for the study design. The rest thought that the 
proposed design should at lease be field tested before funding is sought, 
and most suggested substantial revisions be made before the field testing. 
It is very important to note that all but one of the reviewers who 
recommended that USCCR proceed directly to solicit funding also strongly 
suggested subsequent field testing and revisions prior to field testing. 
Thus, almost all reviewers urged revisions and field testing before 
implementing the study, and more than two-thirds of them thought this 
should be done prior to seeking the commitment of funding. The view of 
the majority of the reviewers may reflect pessimism about the likely results 
of the field testing, or it may reflect other considerations. 

NARRATIVE COMMENTS 

The following excerpts were selected to reflect each reviewer's opinions, 
focusing primarily on those comments to which they give the most emphasis 
and which are not clearly indicated by their rating of the study design. 
Some additional comments have been excerpted which make important but not 
particularly emphasized points; these will be preceded by [N.E.] . 
Specific sugt^estions for improvements will be excluded, but not comments 
about the need, or lack thereof, for improvements, or the ease or 
difficulty of making improvements. 

A summary of the narrative comments is: most reviewers felt this design 
was an improvenent over previous desegregation research, almost all felt 
that there were many aspects which could be improved, and some felt that 
even if the design is improved substantially it will not be as useful for 
policy decision making as implied by its authors. 

DESEGREGATION AND MINORITY STUDENT EDUCATION EXPERTS ; 
FAIRFAX 

"The Design which has been presented to the Commission is essentially a 
study of biracialism in public elementary and secondary schools and is 
not a study o£ deso{,'.regation. In its present form, it would compound the 
current confusion surrounding an historic, internationally significant and 
highly misunderstood phenomenon and will not contribute to the enlightenment 
of our citizenry, which is so desperately needed." 



"The so-called desegregated schools to be studied include what appear to 
me to be: (1) biracial schools in districts which may still be so^^rcj^ated 
by current legal standards, (2) biracial schools in desegrej;ating districts 
which may or may not be desegregating under plans wliich meet minimum court 
standards, (3) biracial schools in desegregated districts." 

"The phenomenon to be studied should be the desegregated district 
(because that ir the unit which the courts deal with and the design 
should define as 'desegregated* schools only those which are in desegre- 
gated districts). I would not oppose scudying segregated schools in 
segregated districts or segregated schools in desegregating districts, 
but they should be clearly identified as such. If the design studies 
biracial schools in districts which are not desegregated and calls them 
desegregated, it will compound the current confusion," 

FOSTER 

"The greatest weakness of the methodology. lies in the Instrumentation. 
Hundreds of unvalidated items are proposed, ranging from Likert scales 
to nominal scaling of obviously continuous constructs (e.g., the 
dichotomous scaling of attitudes). Do the items conform to the APA guideline 
What formal steps have been taken to validate them? When the same instru- 
ments are used with different racial and ethnic groups, it is important to 
illustrate that factor patterns are similar across groups. No mention of 
any such intent has been made. With such a mixed bag of instruments it is 
critical that pilot work be incorporated." 

"Comparing segregated and desegregated schools on the ^impact of desegre- 
gation' is an idle dream. Defining a segregated school as over 90 or 
under 25 percent whine is (admittedly) arbitrary, but it is more serious 
than that. It is completely cra^.y. A school can only be defined as segre- 
gated or desegregated in the context of the school district of which it 
is a component part. Research that does otherwise immediately misses the 
point." 

"Abandon efforts to develop a study that will Influence public policy. 
The latter is not based on social science research nor are the courts apt 
to be influenced to any great extent by such results. Furthermore, there 
will never be any research agreement anyway on policy matters." 

••initiate a heeds assement' from potential research consumers. There 

is little point in spending all this money unless there are a substantial 

number of serious consumers who have a problem/' 

WATSON 

"In general, I think the proposal is a sound one; it does need consistent 
and effective monitoring and will probably require intervention by the 
Civil Rights Commission as the study proceeds*" 
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"I would strongly recommend that the staffing of the research team include 
blacks and Spanish-speaking people, both Puerto Rican and Mexican Americans. 
Unlike Rand's recommendation, I would .'strongly urge that wrltLen 
questionnaires be prepared in Spanish as well as linglish." 

"I seriously question some of the assumptions and conclusions Pand made 
about the attitudes and beliefs of Spanish-speaking people." 

MERCER 

••a tremendous amount of work has obviously gone into the proposal and 

it is greatly improved since last spring. The design is clearly specified 

and coherent.*' 

"My reactions are spelled out on the accompanying sheets which are 
general respouses to the questions with additional comments. The design 
is very sophisticated but I still do not believe that a survey approach 
will provide any definitive answers tu the critical questions about the 
processes which characterize effective desegregated schools. My vote 
would be to cut out all the peripheral studies which make the proposed 
'projects^ an administrative monstrosity and zero in on developing more 
fully the experimental design which the Rand people themselves present as 
their preferred option*" 

"The experimental design would need further work to develop the details. 
This was probably not done because the Rand people saw little possibility 
of that design being funded. In my opinion, it would be feasible. 
Multi-racial schools are anxious to try promising new programs." 

[N. £• ) "My own experience with a small data file of about 200 
desegregated schools suggests that there is so little "between school* 
variance in programs and it is so difficult to ascertain what programs, 
if any, a school is actually implementing that there are almost no 
•relatively unambiguous answers to some questions' which are likely to 
emerge from the non-experimental design option." 

"The response could be even worse than Coleman had. The climate of 
opinion about research has changed since his study and educators and 
citizens in desegregated districts are particularly wary of anything 
that will make waves. I personally believe it is possible to secure 
cooperation, but it takes work, work, worki" 

[ N# £•] "The tests are very short and no information is given about them. 
For example, the rank ordering of ethnic pictures, a measure used in the 
Riverside study, creates many problems in scorinj: and I would not recommend 
Its use in a larger study in the format used in Riverside. To rank one's 
own group high, one must rank other groups low," 
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••We are in a period now where there is much resistance to some of the 
types of questions posed in the various instruments," 

"My overall reaction to the focus of the design is negative. I find it 
difficult to reconcile the Commission on Civil Rights spending the bulk 
of $9 - $14 million for a series of studies to ascertain if the law of 
the land (i.e., desegregation) is working rather than spending this 
amount for studies which would show it can be made to work." 

••Another major objection is the omission of any major consideration of 
the push-out problem, majority group violence against the minority 
group, or firing of minority staff," 

••The promise to provide the new data to other researchers and scholars 
is excellent, and I hope this will be made a part of the formal contract. 
Also, the fact that the Commission may at any time intervene in the 
conduct of the study is good. I hope thnt the Comriiission will provide 
sufficient monitoring staff to really stay on top of the various studies." 

"I would also add that any policy task force assembled to oversee this 
study should have adequate representation." 

DISTINGUISHED SOCIAL SCIENTISTS : 

TYLER 

••Hence, the evaluation of desegregated schools in terms of the extent 
to which children are learning what they are being taught in school 
cannot be done by currently published achievement tests. New instruments 
are necessary." 

"I doubt the feasibility of random assignments of school innovations." 

"A sertond major comment on the instruments is the number of questions 
Included that state laws in California, and perhaps in other states, 
prohibit asking on the grounds of invasion of privacy. Where state laws 
may not prohibit asking these questions, many schools regulations or tacit 
policies effectively piohibit them. This issue should be dealt with before 
the detailed research plan is approved. Among the kinds of prohibited 
questions are; how the child feels toward other children or adults; 
Items about family wealth or poverty, such as appliances in the home, 
autos owned, whether the man in the house is the child •s father; questions 
about the attitudes of parents; etc. 
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AHMANN 

••One does not over-simplify the primary purpose of the research clesip,n 
by stating that it is focused on the generation of data for policy makers. 
This point is emphasized repeatedly and is to In* commended. In contrast, 
however, one finds that no clear statement is incltided in tlie researcli 
design witli regard to the questions which policy makers wish to answer. 
Questions proposed are evidently hypothetical ones based upon an intelligent 
review of the state of the matter today insofar as desegregation of schools 
is concerned." 

•'Evaluative research will succeed if it is based upon the questions 
raised by the users of the data rather than the questions raised by 
those who propose to produce data." 

••Evidently the instruments to be used in the cognitive domain will be 
the traditional standardized achievement tests* Attention should be given 
to the question of objective-referenced and criterion -re ferenced tests as 
used so frequently in areas like the National Assessment of Educational 
Progress. Furthermore, the definitions of achievement should be broadened 
to be sure that areas other than the basic skills are brought into play.'' 

In* E.l "Those who design the study tread rather freely, and in my 
judgment, rather carelessly over a number of osychometric problems when 
designing the proposed instruments to be used. Under no circumstances 
should these instruments be used without careful pretesting coupled with 
item analysis and associated realiability and validity investigation." 

CRONBACH 

•'You will see that I have a definitely favorable opinion as to the value 
of the screener study, the intensive case studies, and the intensive 
study of the newly desegregated schools. I am firmly opposed to the 
experimental study. With regard to the 'longitudinal* and 'student' 
panel studies the technical issue of the adequacy of the matching scheme 
weighs very heavily in my thinking. I judge Rand to be wrong, but I am 
sure that this is an area where they have qualified staff and the difference 
between us probably needs further thought to figure out just why we are so 
divergent. I certainly am not prepared to endorse those studies as presently 
designed." 
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"The most troublesome Issue, In the sense that much hangs on It and 
that the pro-and-con arguments arc hard to orRanl;<e, is the •qunrt(»L' 
design. If that Is as vulnerable, as I think it is, aln^osi all the otlier 
technical decisions have to be reworked, assumin;.; that this program of 
quantitative studies is to be mounled at all# As I sec il, it is 
inappropriate to mount a grand psetido-controlled study unless we have 
confidence that rigorous inference is indeed possible. I do not think that 
one can make a rigorous inference about the impact of a certain type 
of desegregation, in a certain category of school systems, unless the 
sample is truly representative of that category and of that type. The 
attempt at post hoc matching seems to me to guarantee that samples are 
not representative of categories. This threat tc inference would not be 
serious if one could make the assumption that the basis for matching 
(essentially, geographical contiguity and comparability of demography 
in census tract) accounts for all (or nearly all) of the causally relevant 
variance. But there are community characteristics and political events 
that caused one neighborhood to desegregate while the second neighborhood 
did not, or that caused the adoption of different plans. The system of 
matching does nothing to account for these antecedents, and so all comparisons 
are ambiguous.*^ 

"The second pervasive issue is the attempt to generalize over ill-defined 
'types,' under the assumption that a gross label characterizes a useful 
treatment variable that could be written into a policy recommendation. 
It may be, for example, that 'voluntary' desegregation plans differ so 
radically among themselves that an attempt to generalize over that range 
Is inappropriate. Certainly it is true that 'schools with a high degree 
of parent involvement' may have considerable dissimilarity in the character 
and motivation of that involvement. I remark elsewhere on 'bilingual- 
blcultural' and 'extracurricular program' as labels that imply a homo- 
geneity and hence a causal relevance that probably will not actually be 
found." 

"As usual in this document, the section on analysis is superficial 
and injudicious." 

[ N. E.] "The sample of some 200 districts is large enough to assess 
overall zero-order correlations with enough precision. As we move into 
multiple regression and path analysis we become shakier. Also, if we 
entertain the idea of different regressions within the subject that had 
court-ordered dcsegregation--and we had better--we drop down into a 
sample, size where the signal-to-noise ratio is distressing." 
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"This comes to . very major l.sue thnt I didn't touch on in the que.uon 
„I<re-?te whole matter o£ research management, which becomes crucial x. 

undertaking a sparge scale and as interconnected a.s is being proposed. 
I really th nV the e fs no precedent in the applied social so ences. I 
« y iilh "ike the idea of data collection by one -3°;^ -Pf^^''-'/, 
rontracfor with separate contracts for analysis and provision tor a 
Too^ d if free-lancing using the data bank. But the P""-= ° . 
loordinatlon are Immense, and there are serious "°^f^.^Sai-t getting 
the most competent minds to work on contract on so^^^ 

ri:c:r:f1a:ra:e"u™m:J\n''thrir"ur;rclaim on public funds." 

"It seems to me crucial that the Rand group who conceived the project 
b 'the m:nr8e::nt team to hold it "gether-given their thought and 

="::^tier'^:u:rirroi;:tert 7<^:x rit ii r.iisJo.., 

S they had an expensive (and Inefficient) free hand to redesign 
it to the ir taste." 

iii:ri;t airc^^d^tr^trf^^rtLp^ 
i'p-XiiirSn—^^^^^^^^^ - - 

crucial point in evaluating feasibility. 

[ N E 1 "A major technique relied on to nssess intergroup attitudes 
r uAa fL nurtures test-wcrries me. Such a test has been often 
^Ll rrprute y^u Hlth Negro and white stimuli. But how to represent 
Puerto Rican and >■ .lean-American children unambiguously withou 
ra'ppro^pilate ste, otyping 'he pic turesV Puerto .icans are t^ 

pretested," 



ROSEN SHINE 



"The message I send is that you cannot expect definitive answers from 
.1 Ln?nv.n?orv study The major problem is instrumentation, and most 
Tf Th frstruLrt:'!; tSis .tidy Le exploratory ^--^^^/^f/J^^^:^, ^ 
not been used before or have been used .In one or two studies with varied 

results." 
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"I do not recommend going immediately into such a study unless there 
is a possibility of differential outcomes. That is, it* one can expect 
that the pretests (across schools and school districts) will piobably 
account for 90 percent of posttest variance (certainly probably on the 
cognitive measures) or more , then thvre is little room for school 
policies and practices to account for the remaining variance, or, 
differences between schools will be miniscule. 

"Expressing things as regression coefficients does not relieve the 
problem of size of effect. Unless there is good reason to believe that 
there will b6 effects, I see no reason to begin." 

"I think the form of' this survey is excellent; I think that the substance- 
expressed as questionnaires, outcome measures, interviews, and hypotheses- 
is weak. (Examples of weak substance are the student questionnaires on 
individualization or the self-concept tests--both of which don't represent 
the best guesses of scholars in the field. Nowhere is there the detailed 
support for instruments that is given for design)." 

"I claim that the substance is not yet ready, and it is naive to initiate 
this study with such inadequate substance." 

"The analysis of problems in using the achievement measures (pgs. 158+) 
is excellent. NB: If all these difficulties exist with achievement 
tests, they exist even more fully in the unexplored area of non-cognitive 
tests. How, th/jn, can one justify selection of non-cognitive tests which 
have usually been used in a single study.? The non-cognitive measures 
are not ready, to date." 

"Before one invests one million '(to say nothing of ten) in this 
experimental design, one would like some reasonable assurance that the 
results will have meaning. However, the results to date on research 
such as this are not promising." 

"The research on 'it works ' --AIR ' s study of successful school 
innovations has been that innovations are seldom successful accross two 
or more years. Why they are not successful is unknown." 

"The variables to be studied--multiethnic curriculum, individualism, 
and school activities are extremely questionable as appropriate 
exemplars of 'sucessful.' Furthermore, the methods for studying these 
are pocr--multiethnic seems nothing more than having minorities represent! 
in textbooks; individualization means ability to move around the room. 
The literature must be richer than that in identifying characteristics 
of successful schools — if it Isn't, forget the study." 
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ASTIN 

"It was somewhat difficult to evaluate the appropriateness of the 
research design because the general goals of the study are not entirely 
clear." 

"One area of possible significance which gets virtually no attention 
in either the theoretical treatment or the instruments is the impact of 
desegregation on the p«er group behavior of young adults. One very 
important area of peer group behavior is dating," 

"Another problem. is the obvious racial stereotyping that students are 
encouraged to engage in. Again, there must be a more subtle way to get 
at stereotypes rather than asking students to respond yes or no to the 
question, 'Are they dumb?'" 

"In looking over the instruments for school personnel, I find that many 
of the items designed to get at racial attitudes or attitudes toward 
desegregation are much too transparent. Again, some care should be 
devoted to attempting to phrase items in more reasonable terms and to 
uncover more subtle differences in attitudes than could possibly be 
revealed by such obvious questions. In short, I would suggest a very 
careful review of such items by people experienced in developing 
measures of attitudes. 

DEUTSCH 

"I have serious doubts about the' political and ethical feasibility of 
such research. There has been much resistance from schools and parents 
to questionnaires which contain items dealing with racial attitudes. 
While this resistance may be poorly conceived and short-sighted, it 
exists nevertheless; such resistance would seriously interfere with the 
possibility of implementing the proposed study designs. If this study i 
conducted under governmental regulations which require 'informed consent 
of the students (or of their parents) and of others who are to answer 
questionnaires, I suspect that the refusal rate would be high. With 
these problems in mind, I suggest that it might be advisable to conduct 
a feasibility study to determine how much cooperation from schools and 
parents is likely, before going ahead with the studies proposed," 

"The knowledgeability and technical competence of the au*:hors of the 
proposal are impressively high. The proposal, however, suffers from 
an ambiguity with regard to the policy questions to which it is 
addressing itself. The proposed research is very much concerned with 
trying to formulate the policy questions as well as trying to find the 
answers. My own preference would have been to separate more sharply 
the formulating stage and the answering stage. Thus, I think it would 
be helpful to start the research with a series of surveys of relevant 
categories of 'policy makers '--federal judges, city officials, school 
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boards, school principals, teachers, parents, interested civic groups, 
etc.— which would try to identify the specific questions most salient 
for each category. I am confident that the proposed research will 
answer nany of the questions that such categories of people might 
want answered, yet it would be reassuring to know in advance that the 
research is addressed to questions that the potential users of the 
research are concerned about," 

MERTON 

"Nevertheless, the record is reasonably clear that the COLEMN REPORT 
did not have direct, official and therefore readily visible effects 
on the formation of comprehensive policy dealing with educational 
opportunity." 

"...the authors of RAND DESIGN assume that the absence of significant 
official use of thi COLEMAN REPORT is wholly or mostly the result of 
liaiitations and inadequacies in the research design and execution. 
...1 happen to think that... it is not the case that 'good research 
leads directly to 'good public policy.'" 

"In any case, when the question is raised as to whether RAND DESIGN 
can be expected to yield 'relatively unambiguous answers to some ^ 
coherent sets of important policy questions about school desegregation, 
this seems to me to underestimate the gap between social research 
findings and policy formation." 

"One major source of that gap is the multiplicity of consequences of 
any public policy for the diverse sectors and groups in a complex 
society It is the rare situation in which a proposed policy can be 
shown to benefit all sectors and groups and to do so equally or at least 
equitably, as seen from the perspectives of the se diversely affected 
components in the population . In short, the formation ot public policy 
Is inescapably a part of the political process. That means th^t diverse 
group interests, values and belief-systei-s will affect (1) the reading 
of the so-called 'relatively unambiguous answers' and, even^ more , 
(2) the policy- inferences to be drawn from the information." 

"That gap is there even in the case where the research is quite unam- 
biguous. But even the most optimistic of social researchers can 
scarcely describe the massive array of research studies incorporated 
In RAND DESIGN as apt to lead to thoroughly unambiguous results. The 
great bulk of results--even in the proposed experimental part of 
RAND DESIGN--are bound to be of a kind allowing for diverse interpre- 
tations so far as implications for policy are concerned." 
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"In suggesting all this, I am not downgrading the hard thinking lliat: 
went into RAND DICSIGN.. T am suggesting that its authors (and perhaps 
the Commission on Civil Rights itself) may have an overly optimistic 
image of the connections between social research results and large- 
scale public policy." 

"In short, I suggest that at the best, this research program will pro- 
vide findings, ideas and tentative interpretations which can bo taken 
into account and turned to account by the diverse groups and interests 
most directly concerned with school desegregation; that local policies 
will be far more influenced by these concerned groups than by the 
inferences drawn frbm the research by social scientists or by policy- 
makers in Washington; that all manner of 'concessions', 'compromises ' 
and 'adaptations ' of conclusions drawn from the 'policy-relevant 
research' will occur during the political process; and it would be 
unwise to get up one's hopes that unequivocal research findings will 
yield unequivocal policies that will be adopted and translated into 
practice. If thec^e more modest aspirations are what the Commission 
on Civil Rights has adopted for the proposed research program, then 
this will not be still another disappointment for those who had 
persuaded themselves that solid research findings will lead to indicated 
policies and that implementing these policies will in turn lead primarily 
or only to desired consequences." 

PETTIGREW 

"The strongest part of the proposal is its conception of a basic data 
bank from which a variety of studies by numerous investigators could 
be spun-off. This idea is a widely shared one now in the social science 
community, for we have learned from our mistakes of the past two decades 
that countless little studies, each with their own too-small data bases 
and unique instruments, do not add up to what the nation needs and 
rightfully should expect of its investment in social research." 

"The next strongest part of this report is its emphasis on process. 
This is a favorite emphasis of mine over the past decade of writing on 
the subject, so I naturally think it i? long overdue. Indeed, my 
reservations about the design have to do chiefly with the fact that the 
report does not wholeheartedly devote itself to process, but insists on 
retaining aspects of the old Colenian Report emphasis on achievement scores 
comparing desegregated with segregated schools either at one point in time 
or over a too-short period of one year. At many points the report 
rightfully and persuasively argues against this focus of the past} yet it 
nevertheless builds in a dubious type of sampling for segregated schools 
and calls for one-year sample achievement effect results of desegregation 
V8« segregation before the basic process data are developed to 
serve as mediators of the effects in the analysis • My major recommendations 
tot change involve the removal of this repetition of a strategy that we 
know leads to confusion and policy irrelevant findings*" 
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"Finally, in the Interest of coordinating this effort with previous and 
current work in the area, I believe more explicit attention should be 
paid, i. e., to the repetition of successful itunis in othur stnclU-s (some 
of this may well have been done, but is not indicated in the report)." 

WIISON 

"I am strongly in favor of this proposal: a good longitudinal study 
is much overdue; Grain and his colleagues at Rand are well-qualified 
to carry it out. I very much hope the experimental option will be 
retained." 

"1 have one suggestion for modifying the design. Almost all extant 
research on this subject, and the present design, involve the collection 
of individualistic data by questionnaires or interviews aimed at single 
respondents. Such an approach will miss the interaction effects of people 
responding to each other in integrated and non-integrated settings." 

"I propose that in a significant number of schools, an interviewer be 
assigned the task of meeting with small groups of all-white, all-minority, 
and integrated pupils to discuss with them their attitudes toward 
learning, the school, each other, and non-members of the group (especially 
non-members of different races and cultures). The groups should be 
'natural' groups--i.e. , actual peer groups or play groups. Strict 
sampling procedures are not vital nor is precise coding of all responses. 
The object would be to get the groups to explain their feelings and 
actions about and toward the schools and other pupils," 

FEDERAL OFFICIALS RESPONSIBLE FOR DESEGREGATION RESEARCH ; 
RIST 

"I think one of the most basic issues of the entire report is an episto- 
mological one, that of defining terms and defining problems. Many 
Important terms like 'desegregation' or 'segregation' are treated as 
statistical realities, regardless of whether in the context of the actual 
school settings, the interpretation might be different. Statistical 
realities are not always cultural realities and this report does not take 
account of this possibility." 

"Finally, I think there is a hiatus between the expressed goals of the 
research and the methodology set up to gain answers so as to achieve 
thorn. If one wishes to find v-zays ao as to improve integration in 
American society, then that is a very different topic than learning how 
well it is or is not doing with respect to segregation, The research 
may be able to answer the latter, but cannot as presently 
organized answer nhc fortnet, I think it is also a critical 
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question as to whether we should spend 15 million to find out the answer 
to the latter. I am not S'«re with the 'grab ba^' approach to r.he colU-ction 
of data that one will necessarily find in all of It the keys to unlock 
how to improve the current situation. The brief mention of locating 
Schools that are doing well and studying them will not help if the 
questions are kept the same and the approach is still one of comparison 
to segregated education. It is one thing to collect much data that will 
be interesting, it is another to address oneself to issues of social change 
and how iihat change car be enhance'' in what one believes is a constructive 
direction." • 

"It is as if we were out to again discover the wheel. There appears to 
be little sensitivity to much of the present research, particularly at 
the classroom level, on both cognitive and non-cognitive outcomes of 
desegregation. An alternative strategy would be to build on what we now 
know." 

YORK 

"In the ETS study we emphasized successful outputs in developing the 
RFP, in part because of the weak program effects found in the NORC 
report. The intent was to find good candidate exemplary desegregated 
schools based on output measures (statistical adjustments for SES 
differences, etc.) and then conduct a crass empirical approach to find 
the causes of success (broadly classified as specific programs and/or 
school practices and policies). This seems to me to be a more productive 
approach than Rand proposes. The fact that we are having problems in 
finding stable exemplary desegregated schools based on outcomes suggests 
that you might have similar difficulties if you should use our type of 
approach, but the Rand approach would seem to make these problems even 
more serious and more likely. The difficulties in finding stable 
successful schools should not be surprising to Rand after they published 
the excellent piece by Klitgaard, A Statistical Search for Unusually 
Effective Schools ." 

"It has been our experience that colinearity problems in educational 
research are so widespread that they will raise serious analytical 
problems even in the best experimental or quasi-experimental design 
that Rand is likely to be able to produce. I don't intend this as a 
surrender but as a sign that any RFPs issued should emphasize analyses 
methods for dealing with these problems." 
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"A different type of problem is the one created by excessive data collection 
burdens on the schools. School district resistnnctj has been growinf^, 
over the years although v;e have taken several steps that si^em to bo 
helping to reduce the problem. However, the Rand basic study plus the 
supplementary studies will pose a pretty substantial burden ou many o£ 
the 1,600 schools. Since the sample design includes several schools per 
district (and should, to reduce noise for the statistician if not for 
others involved) the probability of a district facing an especially 
heavy data collection burden in at least one school looks very high." 

"It seems to*me that one contractor-- the data collection contractor-- 
should be either a prime contractor or have any coordinating and conference 
scheduling responsibilities for all contractors that are not the respon- 
sibility of the government. A 'separate 'management group' contract 
(II, p. 221) seems to create an extra bureaucratic layer and lessen the 
authority of the data collection contractor (who must be able to exert 
strong direct pressure for delivery of instruments on time and in a 
format consistent with the overall data collection and data processing 
specifications) and interfere with whatever government monitoring 
procedure is developed.'' 

LUCAS 

"One important set of policy variables is systematically neglected-- 
these are cost variables--innovative programs, training, public relations, 
etc., costs $. Moreover intra-school budget decisions may make a 
dif ference-e .g. , how modern and complete is equipment in the chemistry 
department. Also, one variable which performed in the Coleman study- 
Teacher verbal and other skills is missing." 

"Budget decisions are important policy instruments. No decision maker 
Is likely to act on recommendations for program changes to make inte- 
gration work without knowing (a) how much these cost on the average 
(b) the threshold of investment required to make the programs meaningful." 

"It takes inadequate account of the evidence that situational constraints 
rather than attitudes seem to predict (1) the move from desegregation 
to Integration and (2) subsequent changes in attitudes. Coercion, 
financial incentives and financial constraints (e.g., on white flight) 
are neglected." 

M. SMITH 

"Concentrate on socio-political analyses of how to make desegregation 
work peacefully. Almost all the work on how to make a 'desegregated' 
school into an 'integrated' school is poor/' 
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"Another modified planned variations desiRn is fine in principal but 
a disaster in practice until we learn something more about how children 
learn and behave in schools. To suB^.tJ^t, for example, that 'individualized 
Instruction' is a single coherent approach is very naive," 

"Your costs will be much greater than Rand estimates if you carry out 
the whole study--get your estimates from Follow-Through evaluation 
rather than from the EEOS and related studies (you might check with 
Carl Wisler or John Evans)." 

Figure out precisely which policy questions you wish to answer from the 
Federal level, for the State level and for the local level. Generally, 
I suspect those questions need only descriptive studies to provide 
answers-«not studies designed to answer causal connections." 

LOHMAN 

found myself thinking, as I tried to absorb the meaning of 900 pages 
of print, that if the Civil Rights Commission is willing to accept the 
Rand perspective in approaching desegregation as a phenomena of study, 
and if they can support the general methods proposed to study this issue, 
then at least 70 percent of the studies should be considered worthwhile 
to support." 

"On the other hand, if you formulate the desegregation issue with a 
different set of assumptions than those adooted by the Rand team--a 
different set of studies and priorities will emerge, and only ten 
percent of the Rand design would be useful. In short, if the second 
view is adopted a new team x^ould have to start again." 

'*How does one describe these two different approaches, and what is tr.y 
recommendation?" 

"Position A--Desegregation as a social science problem worthy of 
scientific analysis and carefully controlled experiments." 

•'Position B--Integration as a desirable goal to achieve in American 
Education. " 

"It should be apparent that strategy A represents the approach adopted 
by the Rand research team. It is a respectable strategy for understanding 
what the current national experience with desegregation is like. But 
It will offer very few, if any, solutions to the problems--multiple 
and divcrsc--which are faced by school districts across the country. 
Strategy B focuses upon finding solutions to problems. Its answers 
will rarely be universal in application; and it will necessarily be 
selective about what problems it can reasonably offer any quick answers 
to In the next six years." 
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"If the Corr^lssioncrr, arc willing to accept ntrato^.y A, l1u..n they should 
accept the Rand dcsign-if they accept strategy I',, then they should 
.reject the Rand design.. It's that simple. 

NOTC: On the evaluation questions H to #9. I)r Lohman rated ^e-J^J;" 
twice once in respect to "position A" and once in respect to position 
J" ihe averages of the two rating, were used for the shown .n 

attachments B and C; Dr. Lohman rated position A one-half to one full 
rating abote the rating which I have indicated for these questions, 

and rated position B one full rating below the indicated rating. 



MORNELL 

"Given this, overwhelming document, it is important not co lose sight 
of the simple questions with which this evaluation must be concerned: 

1. TO what extent is the design as a whole appropriate to the intent 
of the U. S. Commission on Civil Rights? 

2. TO whar. extent is the design as a whole efficient, realistic, and 
likely to achieve significant results? 

"The design basically proposes a massive data bank from which social • 
scienti fwith a mult'ituSe of interests will draw in their effor s to 
develop significant conclusions. To make -the bank as ^"elusive as 
possible, no variable, hypothesis, or question which might be "lated 
to desegregation knowingly has been excluded. Further. ^^"^"f^^ 
research techniques have been proposed, although survey research obviously 
is dominant,'' 

"The result of this approach is a proposal that touches upon 'everything' 
in the field of schooi desegregation, with little effort to « ate biases 
cLarly. define precise objectives, and develop «P-^^^/^^>'P°f^7^^\°'3 
models. (The 'analytic models' which appear as part of the design are 

fac . no more than tables of po3Sible relationships ^^T^^" -^^J^^^^f^ 
which are to be correlated.) However, this attempt to be o^^^^^^^^^ J;^ 
often contradicted by the value-laden statements which appear througuout 
the pa^er leaving the impression thatthe design is a political document 
intended ^o appeal to the n^ny points of view on desegregation and the 
many interests of research scientists and funding agencies. 

"This approach is justified by the stated assumption that there is little 
aprecment as to which policy issues are most important-an assumption 
Shich is .'.learly false, especially as far as the Cotmission is concerned. 
It is also based on the stated assumption that no single study could 
iuccced settling the outstanding policy issues-an «f ""^P^^^" ^^^^^^'^^ 
clearly tr.io if the first assumption determines the design, ^^o'^'^^*-^^ ' . ^ 
even given these assumptions and this design, there is no way to predict 
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If tho study will load to such conclusions. A p.roatcr por.«iblUty is 
that tho study will load Lo a host of inconclusive, contrnd ic lory . and 
insignificant' findings, ^ny o£ which may bo usod to attack dosugru^at .on." 

LYONS 

"Although Commission staff and consultants repoatudly emphasi^^ed the 
need to analyse the racial composition of educational personnol within 
schoo districts and individual schools, and its effect on the various 
school outcome criteria, tho final design virtually ignores his critical 
^ttei. This omission is appalling. School desogregation U^eratuce is 
replete with theoretical assumptions and research findin, •. pertaining to 
the effect of staff racial composition on education and race relations. 
Moreover, the problem of demotion and dismissal of minority teachers 
and administrators in some desegregating districts is of grave concern 
to the civil rights community." 

"This omission cannot be cured adequately by 'quick ^" V^"'^ ' "^^'^hnL 
to exisUng instruments. The determination of data needs and the methods 
to be employed in securing necessary data should fo low; 1) a survey ot 
existing research and data resources. 2) conceptualization of probable 
relationships between staff racial composition and key educa ional and 
race relations variables in the overlapping systems of school and 
community, and 3) formulation of testable policy-relevant research 
hypotheses/' 

"Another major problem of the design is the lack of attention given to 
the matter of school discipline. Discipline is a critical concern of 
wSitrnd minority parents, of tax-paying citizens, and o students and 
school personnel alike. Discipline is signif cant not on y b-ause it 
is controversial, but also because it strategically ^ ^ .ff^^^ 
range of educational processes and outcomes encompassed by the design. .. . 
The design is mute on such obviously important subjects as student 
Suspensions and expulsions, the character and naturo of discipline codes, 
and the manner in which they are enforced. 

"Another major shortcoming of the design is the failure to specify 
a^es between the .^jor studies and the substudios -^^/f ' , 
often refers to the survey data base as the resource for validating 
potential findi.,s of specific substudios. Such re erences are casual if 
not cavalier, and largely ignore the critical questions of data 
comparability and control." 
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FEAGIN 

"The proposal is particularly weak in two way«: (1) it clous not make 
clear the probleni of defining and studying 'de?]eRr(»yated ' schools, 
some of v/hich are desegregatod because they are in a frinj^e area, some 
because they are token schools in a Renerally suRregated school district, 
and some because they are in completely desegregated districts; (2) it 
does not focus on classroom interaction*" 

"More attention to the issue of "desegregated ' schools. How many are 
there? What categories do they fall into? Are there enough truly 
desegregated districts for an adequate sample? More attention to the 
timing issue. What does it mean to come into the desegregation process 
at various points in time?" 

"Again, some very important questions have been slighted. But some 
important policy questions will receive attention." 

ORFIELD 

"I doubt that any study could provide unambiguous ansv/ers [to policy 
questions]. The benefits will probably come from a more adequate 
understanding of the process, an indication of clearly useless strategies 
and some general showing of areas of possible high return." 

"(There is a need for:] a more competent section on Chicane and Puerto 
Rican educational problems and their interaction with desegregat ion-- 
and development of appropriate instruments; a stronger concentration on 
elementary grades, with particular attention to impact of beginning 
school in desegregated or integrated settings--both in big study and in 
study of newly desegregated schools; a control group of segregated white 
schools in the quartets." 

SMYTHE 

"My impression of the design is that it tries harder than it did at first, 
but is a remarkably weak piece of work for a final draft, even given the 
problems involved." 

"Unfortunately, I have grave questions about the clarity and objectivity 
of the questions in Volume III. Fuzziness about the meaning of terms 
('Race,' 'unfair*), definitions which conflict with normal usage ('How 
many brothers and sisters do you' have? (count yourself)' '1, only mo'), 
and answers not consistent with the questions ('Would you like to take..* 
Spanish?' Yes, I do take. . tSpanish* ' ) are only some of the problems." 
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"Unfortunately, there are so nuany cultural biases and insensitive 
expressions that there will be resentment of such questions as 'Arc 
they dumb?' (Volunie III, p. A3) in the absence of a corollary 'Are 
they smart?' or allusion to Orientals as 'Japanese' or 'Chinese' without 
adding 'American,' as well#" 

"Do careful case studies of several school systems, identifying the 
Issues 9 procedures, community pressures, etc., as well as outcomes in 
terms of the objectives of the study. (Boston, Denver, Brunswick (Ga.), 
and Miami might be candidates.)" 

JACKSON 

••This design, in its present condition, is superior in many respects 
to any prior desegregation studyj however, it is incomplete and unpolished, 
and will require a substantial amount of work before it is a state-of- 
the-art product. In addition, even if the design is improved as much 
as is possible, given our present level of social science research 
technology, there are a number of difficulties which will remain and 
which will preclude the making of many firm policy oriented inferences 
from any study of school desegregation." 

"The most important correctable inadequacies of the design are: the 
£ai7.ure to recommend suitable cognitive achievement tests and to measure 
a number of variables which are generally considered important in the 
comprehensive assessment of school desegregation (see Appendix A of my 
narrative comments); the lack of Spanish translations (which the 
Commission agreed to forego in this contract); a moderate number of 
questionnaire items which at least need to be reworded, if not recon- 
ceptualized, in order to be reliable and valid; inadequate sample sizes, 
and sample procedures which may be inappropriate for presently available 
statistical tools; incomplete instructions for the procedures to be used 
in administering the instruments; and some remaining omissions, in respect 
to the major educational concerns of blacks and Spanish origin people." 

"If the above inadequacies are corrected, this study, and any other 
design, will still be seriously limited in making firm inferences about 
most of the policy relevant aspects of school desegregation." 

••Discussion in volumes I and II of the Rand, design clearly indicates 
non-experimental longitudinal designs are intrinsically quite limited in 
their ability to yield firm policy relevant inferences about complex 
phenomena." 
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"The most decisive statement about the inadequacies of a non-oxper mcntnl 
longitudinal design i« on page 157 of volume TI.. Many ^J;^^^" 
effects of school desegregation have been longi Ltidinal . which dots 
rcDrcsent a considerable' improvement in metnodolo,;v over one-l imo cross- 
"sectional studies such as the Coleman Report; however the more existence 
of a longitudinal study does not mean that probloms of ^P''''^^'^.;^^^ °^ . ^ 
bias are solved, as indicated by the intense controversy over the effects 
of Headstart [which were assessed longitudinally]. 

"The Rand proposal strongly states the methodological superiority of 
an experimental design option." 

"The proposal notes that an experimental design has been considered 
by Lnv persons as socially or politically infeasible but that these 
persons may be wrong. Though I agree that in theory the experimental 
design would be methodologically superior, I believe the Rand P'^opo^^l 
ove looks two serious practical difficulties in implementing ^-^^ a design 
for the purposes of this study. First, if the innovations to be tested 
are likely to work, they probably have to be much broader than indivi- 
dualized instruction,' 'human relations training.' or ^'""1*=^^':'^*=^"^, , . 
curriculum.' They probably will have to involve school practices related 
to student government, discipline, instructional modes curricu u.. 
Fewer schools are likely to agree to the random assinment of a ^road 
intervention than are likely to agree to the random assignment 
and discrete treatment. Second, in all field experiments and Particularly 
in school experiments there is substantial difficulty in controlling he 
specified treatments. The broader the interventions, the more difficult 
the problem becomes. It is relatively simple to train teachers and 
principals to engage in a given set of practices, but it is almost 
impossible to assure that they will fairly consistently maintain those 
practices over the protracted period of time necessary to assess the 
slowly accumulating effects of schooling." 

CONCLUSIONS 

The ratings given in the evaluation questionnaire, and the narrative 
comments provided by the reviewers suggest a wide spectrum of opinion 
aCt the^Rand study design. Dispite the fact that there are areas of 
clear disagreement, there are also areas of clear agreement. There is 
disagreement about whether the Commission should try to implement a 
study based on the Rand design; most of the reviewers believe that such 
a study would be of at least limited utility, but some believe it would 
be useless or even harmful to the USCCR's objectives for ^ 
study of school desegregation. There seems to be fairly strong agreement 
that the Rand study, and perhaps any feasible study, will not produce 
many unambiguous answers to important policy questions, but a "t^J^rity of 
the reviewers also felt that such a study could provide more reliable 
answers than arc presently available to some of those questions and 
would somewhat further our body of knowledge about '^f ^"^^^^ 
Intorrntion. There is unanimous agreement that revisions and "'Ijitions 
ire desirable before implementing the study, but some reviewers believed 
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extensive changes would be necessary and others felt only modc»rate changes 
were repeated by a substantial percentage of the ruviewors, some were 
mentioned by only one or a couple of the reviowurs, and a few were 
contradicted by the changes recommended by other ruviewors. Almost all 
the reviewers endorsed the desirability of field testinj^ the instruments 
before implementing any study, and more than tv;o thirds felt this should 
be done prior to seeking funding for a full-scale implementation of the 
study. 
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Attnchmcnt A 
Evaluation Questions 



Below arc a number of quctions which wc boUovo ■•>'■<= , '° 

consid« wlicn assessing the enclosed study Z^"" 
best iuJjment to answer each question with the multiple .;ho ce 
scale wWc^ is provided-, circle the choice which most closely repre- 
sentt your judgment. The space prnvlded arter each question for 
any clarifying^o-mncnts you wish to ™ke; including comments about how 
any ciari.iyi.ub ^ c fUr. cMiHv desicn which you feel are 

to improve those aspects of the study aesign whj-l-h y 

inadequate. 

1. How well does -the study design focus on the 

processes involved in school desegregation and integration 
which most importantly need to be researched for the 
purposes of sound policy making? 

2. How well does the study design make use of relevant theory? 

3. How well does the study design make use of previous school 
desegregation research? 

4. How well does the study design avoid the pitfalls which were 
experienced by previous large scale education evaluation 
Tfforts (Projec? Talent. Equality of Educational Opportunity 
Survey, National Assessment of Education Progress, Headstart, 
Follow Through, etc.)? 

5. How well does the study design address the major educational 
concerns of blacks, Mexican Americans, Puerto Ricans and 
Anglos? 

6. How well does the study design state questions or hypotheses 
specifically enough to be effectively and efficiently 
researched? 

7 How well does the non experimental design option provide for 
' relatively unambiguous answers to at least some answers to 
at least some coherent sets of important policy questions 
about school desegregation? 

8. How well does the experimental design option provide for 
relatively unambiguous answers to at least some answers to 
at least some coherent sets of important policy questions 
about desegregation? 

9 How well can the study design accurately assess the impact 
^' o? school desegregation if most of the schools in the sample 

remain segregated or desegregated for the duration of the 

Study? 
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10. How well can a longitudinal study nccurnto.ly ar.scss tho. impact 
of school and community variables on studunts if thrsv. 
variables arc not constant in each fiivc^n scliool or conuniimfy 
over the period of the study or if student mobiULy ro-.uUs 

in substantial changes to the student body over the. period of 
the study? 

11. Is the non experimental design option socially and politically 
feasible to implement? 

12. Is the experimental design option socially anu politically 
feasible to implement? 

13. Is the non experimental design option ethical? 

14. Is the experimental design option ethical? 

15. Is there adequate rationale or justification for the sampling 
procedures recommended in this study design? 

16. How well do the sampling procedures generally appear to provide 
needed sensitivity and freedom from biases? 

17. Are the sizes of the various samples likely to be adequate, but 
not unnecessarily large, for the proposed analyses? 

18. The 1966 Equality of Educational Opportunity Study (the "Colemen 
Report") was considered seriously flawed by many reviewers 
because of the poor response rate, particularly of high schools 
(only 397o of the sampled high schools cooperated sufficiently 

to be included in the analyses). How well does this study 
design provide mechanisms to avoid the repetition of this 
difficulty? 

19. Are the items generally precise, complete, unloaded, and written 
at an appropriate level of vocabulavly and syntax? 

20. Do the instruments generally appear to bo likely to yield 
reliable measures of the variables of interest in this study? 

21. Do the instruments generally appear to be likely to yield 
valid measures of the variables of interest in this study? 

22. How well do the instruments cover an appropriate set of 
academic or cognitive skills? 

23. How well do the instruments cover an appropriate set of 
affective variables? 

24. How well do the instruments cover an appropriate ict of 
bchaviorial variables? 

er|c * 86 



25. Do the Instruments Renornlly appear to be unohjectlonable 
to the persons whose cooperation in ndmini.sttir in^ and 
completing the instruments will be nt^c(^ssary? 

26. Arc there at least two measures for most of the constructs 
which are of primary importance to the study? 

The possible response choices were: 

quite well moderately well moderately poorly quite poorly 
strongly agree moderately agree moderately disagree 
strongly disagree 

27. Given your overall assessment of the research questions of 
interest and the present study design, which course of 
action best represents how you feel the Commission should 
proceed with the present study design? 

a. Solicit funding for the non experimental design option 

b. Solic^.c funding for the experimental design option 

c. Pre-test the instruments, and delay a decision about 
implementing the study until the results of the pre- 
test are available 

di Engage in a substantial revision of the present study 
design and/or instruments before making a commitment to 
a pre-test 

e« Abandon the present study design and develop a new one 
with the same or a similar focus, but with different 
procedures i 

£. Abandon the present study design and develop a new one 
focusing on different aspects of school desegregation 

g. Abandon all efforts to do a policy relevant study of 
school desegregation 
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Attachment C 



AVKRACF. t^TING AND RKCOMMKMnKD COUKSflS OF ACTTON OF KACll KKSPONOKNT 



Average 
Rating; 



Rocotmncnclocl 
Courses of 
Action 



J. Fairfax 
G. Foster 

B. Watson 

C. H. Smith 
J. Mercer 



X 

i.i 
2.0 
3.0 
2.5 



d/g 
a/c 
f 
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M.B. Smith 
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M. Deutsch 
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J.Q. Wilson 
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R. York 
T. Lucas 
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1 • quite well . 

2, moderately well 

3, moderately poorly 

4, quite poorly 

Letter 

a. Solicit funding for the 
non experimental design 
option 

b. Solicit funding for the 

experimental design option 

c. Pre-test the instruments, 
and delay a decision about, 
implementing the study 
until the results of the 
pre-test are available 

d. Engage in a substantial 
revision of the present 
study design and/or 
Instruments before making 
a commitment to a pro- test 

e. Abandon the present study 
design and develop a new 
one with the same or a 
similar focus, but with 
different procedures 

£• Abandon the present study 
design and develop a new 
one focusing on different 
aspects of schools desegre- 
gation 

Abandon all efforts to do 
Q policy relevant study of 
school desegregation 
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